<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>36.2. Building Indexes</title>
<link rel="stylesheet" href="dbstyle.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.72.0">
<link rel="start" href="index.html" title="Programmer's Reference Guide">
<link rel="up" href="zend.search.lucene.html" title="Chapter 36. Zend_Search_Lucene">
<link rel="prev" href="zend.search.lucene.html" title="Chapter 36. Zend_Search_Lucene">
<link rel="next" href="zend.search.lucene.searching.html" title="36.3. Searching an Index">
<link rel="chapter" href="introduction.html" title="Chapter 1. Introduction to Zend Framework">
<link rel="chapter" href="zend.acl.html" title="Chapter 2. Zend_Acl">
<link rel="chapter" href="zend.auth.html" title="Chapter 3. Zend_Auth">
<link rel="chapter" href="zend.cache.html" title="Chapter 4. Zend_Cache">
<link rel="chapter" href="zend.config.html" title="Chapter 5. Zend_Config">
<link rel="chapter" href="zend.console.getopt.html" title="Chapter 6. Zend_Console_Getopt">
<link rel="chapter" href="zend.controller.html" title="Chapter 7. Zend_Controller">
<link rel="chapter" href="zend.currency.html" title="Chapter 8. Zend_Currency">
<link rel="chapter" href="zend.date.html" title="Chapter 9. Zend_Date">
<link rel="chapter" href="zend.db.html" title="Chapter 10. Zend_Db">
<link rel="chapter" href="zend.debug.html" title="Chapter 11. Zend_Debug">
<link rel="chapter" href="zend.dojo.html" title="Chapter 12. Zend_Dojo">
<link rel="chapter" href="zend.dom.html" title="Chapter 13. Zend_Dom">
<link rel="chapter" href="zend.exception.html" title="Chapter 14. Zend_Exception">
<link rel="chapter" href="zend.feed.html" title="Chapter 15. Zend_Feed">
<link rel="chapter" href="zend.filter.html" title="Chapter 16. Zend_Filter">
<link rel="chapter" href="zend.form.html" title="Chapter 17. Zend_Form">
<link rel="chapter" href="zend.gdata.html" title="Chapter 18. Zend_Gdata">
<link rel="chapter" href="zend.http.html" title="Chapter 19. Zend_Http">
<link rel="chapter" href="zend.infocard.html" title="Chapter 20. Zend_InfoCard">
<link rel="chapter" href="zend.json.html" title="Chapter 21. Zend_Json">
<link rel="chapter" href="zend.layout.html" title="Chapter 22. Zend_Layout">
<link rel="chapter" href="zend.ldap.html" title="Chapter 23. Zend_Ldap">
<link rel="chapter" href="zend.loader.html" title="Chapter 24. Zend_Loader">
<link rel="chapter" href="zend.locale.html" title="Chapter 25. Zend_Locale">
<link rel="chapter" href="zend.log.html" title="Chapter 26. Zend_Log">
<link rel="chapter" href="zend.mail.html" title="Chapter 27. Zend_Mail">
<link rel="chapter" href="zend.measure.html" title="Chapter 28. Zend_Measure">
<link rel="chapter" href="zend.memory.html" title="Chapter 29. Zend_Memory">
<link rel="chapter" href="zend.mime.html" title="Chapter 30. Zend_Mime">
<link rel="chapter" href="zend.openid.html" title="Chapter 31. Zend_OpenId">
<link rel="chapter" href="zend.paginator.html" title="Chapter 32. Zend_Paginator">
<link rel="chapter" href="zend.pdf.html" title="Chapter 33. Zend_Pdf">
<link rel="chapter" href="zend.registry.html" title="Chapter 34. Zend_Registry">
<link rel="chapter" href="zend.rest.html" title="Chapter 35. Zend_Rest">
<link rel="chapter" href="zend.search.lucene.html" title="Chapter 36. Zend_Search_Lucene">
<link rel="chapter" href="zend.server.html" title="Chapter 37. Zend_Server">
<link rel="chapter" href="zend.service.html" title="Chapter 38. Zend_Service">
<link rel="chapter" href="zend.session.html" title="Chapter 39. Zend_Session">
<link rel="chapter" href="zend.soap.html" title="Chapter 40. Zend_Soap">
<link rel="chapter" href="zend.test.html" title="Chapter 41. Zend_Test">
<link rel="chapter" href="zend.text.html" title="Chapter 42. Zend_Text">
<link rel="chapter" href="zend.timesync.html" title="Chapter 43. Zend_TimeSync">
<link rel="chapter" href="zend.translate.html" title="Chapter 44. Zend_Translate">
<link rel="chapter" href="zend.uri.html" title="Chapter 45. Zend_Uri">
<link rel="chapter" href="zend.validate.html" title="Chapter 46. Zend_Validate">
<link rel="chapter" href="zend.version.html" title="Chapter 47. Zend_Version">
<link rel="chapter" href="zend.view.html" title="Chapter 48. Zend_View">
<link rel="chapter" href="zend.xmlrpc.html" title="Chapter 49. Zend_XmlRpc">
<link rel="appendix" href="requirements.html" title="Appendix A. Zend Framework Requirements">
<link rel="appendix" href="coding-standard.html" title="Appendix B. Zend Framework Coding Standard for PHP">
<link rel="appendix" href="copyrights.html" title="Appendix C. Copyright Information">
<link rel="index" href="the.index.html" title="Index">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.creating" title="36.2.1. Creating a New Index">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.updating" title="36.2.2. Updating Index">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.document-updating" title="36.2.3. Updating Documents">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.counting" title="36.2.4. Retrieving Index Size">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.optimization" title="36.2.5. Index optimization">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.permissions" title="36.2.6. Permissions">
<link rel="subsection" href="zend.search.lucene.index-creation.html#zend.search.lucene.index-creation.limitations" title="36.2.7. Limitations">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<div class="navheader"><table width="100%" summary="Navigation header">
<tr><th colspan="3" align="center">36.2. Building Indexes</th></tr>
<tr>
<td width="20%" align="left">
<a accesskey="p" href="zend.search.lucene.html">Prev</a> </td>
<th width="60%" align="center">Chapter 36. Zend_Search_Lucene</th>
<td width="20%" align="right"> <a accesskey="n" href="zend.search.lucene.searching.html">Next</a>
</td>
</tr>
</table></div>
<div class="sect1" lang="en">
<div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="zend.search.lucene.index-creation"></a>36.2. Building Indexes</h2></div></div></div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.creating"></a>36.2.1. Creating a New Index</h3></div></div></div>
<p>
            Index creation and updating capabilities are implemented within the Zend_Search_Lucene component, as well as the Java Lucene project.
            You can use either of these options to create indexes that Zend_Search_Lucene can search.
        </p>
<p>
            The PHP code listing below provides an example of how to index a file
            using Zend_Search_Lucene indexing API:
        </p>
<pre class="programlisting">&lt;?php
// Create index
$index = Zend_Search_Lucene::create('/data/my-index');

$doc = new Zend_Search_Lucene_Document();

// Store document URL to identify it in the search results
$doc-&gt;addField(Zend_Search_Lucene_Field::Text('url', $docUrl));

// Index document contents
$doc-&gt;addField(Zend_Search_Lucene_Field::UnStored('contents', $docContent));

// Add document to the index
$index-&gt;addDocument($doc);
        </pre>
<p>
            Newly added documents are immediately searchable in the index.
        </p>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.updating"></a>36.2.2. Updating Index</h3></div></div></div>
<p>
            The same procedure is used to update an existing index. The only difference
            is that the open() method is called instead of the create() method:
        </p>
<pre class="programlisting">&lt;?php
// Open existing index
$index = Zend_Search_Lucene::open('/data/my-index');

$doc = new Zend_Search_Lucene_Document();
// Store document URL to identify it in search result.
$doc-&gt;addField(Zend_Search_Lucene_Field::Text('url', $docUrl));
// Index document content
$doc-&gt;addField(Zend_Search_Lucene_Field::UnStored('contents', $docContent));

// Add document to the index.
$index-&gt;addDocument($doc);
        </pre>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.document-updating"></a>36.2.3. Updating Documents</h3></div></div></div>
<p>
            The Lucene index file format doesn't support document updating.
            Documents should be removed and re-added to the index to effectively update them.
        </p>
<p>
            <code class="code">Zend_Search_Lucene::delete()</code> method operates with an internal index document id. It can be retrieved
            from a query hit by 'id' property:
        </p>
<pre class="programlisting">&lt;?php
$removePath = ...;
$hits = $index-&gt;find('path:' . $removePath);
foreach ($hits as $hit) {
    $index-&gt;delete($hit-&gt;id);
}
        </pre>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.counting"></a>36.2.4. Retrieving Index Size</h3></div></div></div>
<p>
            There are two methods to retrieve the size of an index in Zend_Search_Lucene.
        </p>
<p>
             <code class="code">Zend_Search_Lucene::maxDoc()</code> returns one greater than the largest possible document number.
             It's actually the overall number of the documents in the index including deleted documents, 
             so it has a synonym: <code class="code">Zend_Search_Lucene::count()</code>.
        </p>
<p>
             <code class="code">Zend_Search_Lucene::numDocs()</code> returns the total number of non-deleted documents.
        </p>
<pre class="programlisting">&lt;?php
$indexSize = $index-&gt;count();
$documents = $index-&gt;numDocs();
        </pre>
<p>
            <code class="code">Zend_Search_Lucene::isDeleted($id)</code> method may be used to check if a document is deleted.
        </p>
<pre class="programlisting">&lt;?php
for ($count = 0; $count &lt; $index-&gt;maxDoc(); $count++) {
    if ($index-&gt;isDeleted($count)) {
        echo "Document #$id is deleted.\n";
    }
}
        </pre>
<p>
            Index optimization removes deleted documents and squeezes documents' IDs in to a smaller range.
            A document's internal id may therefore change during index optimization.
        </p>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.optimization"></a>36.2.5. Index optimization</h3></div></div></div>
<p>
            A Lucene index consists of many segments. Each segment is a completely independent set of data.
        </p>
<p>
            Lucene index segment files can't be updated by design. A segment update needs full segment
            reorganization. See Lucene index file formats for details
            (<a href="http://lucene.apache.org/java/docs/fileformats.html" target="_top">http://lucene.apache.org/java/docs/fileformats.html</a>)
            <sup>[<a name="id3082041" href="#ftn.id3082041">6</a>]</sup>.
            New documents are added to the index by creating new segment.
        </p>
<p>
            Increasing number of segments reduces quality of the index, but index optimization restores it.
            Optimization essentially merges several segments into a new one. This process also doesn't update segments.
            It generates one new large segment and updates segment list ('segments' file).
        </p>
<p>
            Full index optimization can be trigger by calling the <code class="code">Zend_Search_Lucene::optimize()</code> method. It merges all
            index segments into one new segment:
        </p>
<pre class="programlisting">&lt;?php
// Open existing index
$index = Zend_Search_Lucene::open('/data/my-index');

// Optimize index.
$index-&gt;optimize();
        </pre>
<p>
            Automatic index optimization is performed to keep indexes in a consistent state.
        </p>
<p>
            Automatic optimization is an iterative process managed by several index options. It merges very small segments
            into larger ones, then merges these larger segments into even larger segments and so on.
        </p>
<div class="sect3" lang="en">
<div class="titlepage"><div><div><h4 class="title">
<a name="zend.search.lucene.index-creation.optimization.maxbuffereddocs"></a>36.2.5.1. <span class="emphasis"><em>MaxBufferedDocs</em></span> auto-optimization option</h4></div></div></div>
<p>
                <span class="emphasis"><em>MaxBufferedDocs</em></span> is a minimal number of documents required before
                the buffered in-memory documents are written into a new segment.
            </p>
<p>
                <span class="emphasis"><em>MaxBufferedDocs</em></span> can be retrieved or set by <code class="code">$index-&gt;getMaxBufferedDocs()</code> or
                <code class="code">$index-&gt;setMaxBufferedDocs($maxBufferedDocs)</code> calls.
            </p>
<p>
                Default value is 10.
            </p>
</div>
<div class="sect3" lang="en">
<div class="titlepage"><div><div><h4 class="title">
<a name="zend.search.lucene.index-creation.optimization.maxmergedocs"></a>36.2.5.2. <span class="emphasis"><em>MaxMergeDocs</em></span> auto-optimization option</h4></div></div></div>
<p>
                <span class="emphasis"><em>MaxMergeDocs</em></span> is a largest number of documents ever merged by addDocument().
                Small values (e.g., less than 10.000) are best for interactive indexing, as this limits the length
                of pauses while indexing to a few seconds. Larger values are best for batched indexing and speedier
                searches.
            </p>
<p>
                <span class="emphasis"><em>MaxMergeDocs</em></span> can be retrieved or set by <code class="code">$index-&gt;getMaxMergeDocs()</code> or
                <code class="code">$index-&gt;setMaxMergeDocs($maxMergeDocs)</code> calls.
            </p>
<p>
                Default value is PHP_INT_MAX.
            </p>
</div>
<div class="sect3" lang="en">
<div class="titlepage"><div><div><h4 class="title">
<a name="zend.search.lucene.index-creation.optimization.mergefactor"></a>36.2.5.3. <span class="emphasis"><em>MergeFactor</em></span> auto-optimization option</h4></div></div></div>
<p>
                <span class="emphasis"><em>MergeFactor</em></span> determines how often segment indices are merged by addDocument().
                With smaller values, less RAM is used while indexing, and searches on unoptimized indices are faster,
                but indexing speed is slower. With larger values, more RAM is used during indexing, and while searches
                on unoptimized indices are slower, indexing is faster. Thus larger values (&gt; 10) are best for batch
                index creation, and smaller values (&lt; 10) for indices that are interactively maintained.
            </p>
<p>
                <span class="emphasis"><em>MergeFactor</em></span> is a good estimation for average number of segments merged by one auto-optimization pass.
                Too large values produce large number of segments while they are not merged into new one. It may be a cause of
                "failed to open stream: Too many open files" error message. This limitation is system dependent.
            </p>
<p>
                <span class="emphasis"><em>MergeFactor</em></span> can be retrieved or set by <code class="code">$index-&gt;getMergeFactor()</code> or
                <code class="code">$index-&gt;setMergeFactor($mergeFactor)</code> calls.
            </p>
<p>
                Default value is 10.
            </p>
<p>
                Lucene Java and Luke (Lucene Index Toolbox - <a href="http://www.getopt.org/luke/" target="_top">http://www.getopt.org/luke/</a>)
                can also be used to optimize an index. Latest Luke release (v0.8) is based on Lucene v2.3 and compatible with
                current implementation of Zend_Search_Lucene component (ZF 1.6). Earlier versions of Zend_Search_Lucene implementations
                need another versions of Java Lucene tools to be compatible:
                </p>
<div class="itemizedlist"><ul type="disc">
<li><p>ZF 1.5 - Java Lucene 2.1 (Luke tool v0.7.1 - <a href="http://www.getopt.org/luke/luke-0.7.1/" target="_top">http://www.getopt.org/luke/luke-0.7.1/</a>)</p></li>
<li><p>ZF 1.0 - Java Lucene 1.4 - 2.1 (Luke tool v0.6 - <a href="http://www.getopt.org/luke/luke-0.6/" target="_top">http://www.getopt.org/luke/luke-0.6/</a>)</p></li>
</ul></div>
<p>
            </p>
</div>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.permissions"></a>36.2.6. Permissions</h3></div></div></div>
<p>
            By default, index files are available for reading and writing by everyone.
        </p>
<p>
            It's possible to override this with the <code class="code">Zend_Search_Lucene_Storage_Directory_Filesystem::setDefaultFilePermissions()</code> method:
        </p>
<pre class="programlisting">&lt;?php
// Get current default file permissions
$currentPermissions = Zend_Search_Lucene_Storage_Directory_Filesystem::getDefaultFilePermissions();

// Give read-writing permissions only for current user and group
Zend_Search_Lucene_Storage_Directory_Filesystem::setDefaultFilePermissions(0660);
        </pre>
</div>
<div class="sect2" lang="en">
<div class="titlepage"><div><div><h3 class="title">
<a name="zend.search.lucene.index-creation.limitations"></a>36.2.7. Limitations</h3></div></div></div>
<div class="sect3" lang="en">
<div class="titlepage"><div><div><h4 class="title">
<a name="zend.search.lucene.index-creation.limitations.index-size"></a>36.2.7.1. Index size</h4></div></div></div>
<p>
                Index size is limited by 2GB for 32-bit platforms.
            </p>
<p>
                Use 64-bit platforms for larger indices.
            </p>
</div>
<div class="sect3" lang="en">
<div class="titlepage"><div><div><h4 class="title">
<a name="zend.search.lucene.index-creation.limitations.filesystems"></a>36.2.7.2. Supported Filesystems</h4></div></div></div>
<p>
                Zend_Search_Lucene uses <code class="code">flock()</code> to provide concurrent searching, index updating and optimization.
            </p>
<p>
                According to the PHP <a href="http://www.php.net/manual/en/function.flock.php" target="_top">documentation</a>,
                "<code class="code">flock()</code> will not work on NFS and many other networked file systems".
            </p>
<p>
                Do not use networked file systems with Zend_Search_Lucene.
            </p>
</div>
</div>
<div class="footnotes">
<br><hr width="100" align="left">
<div class="footnote"><p><sup>[<a name="ftn.id3082041" href="#id3082041">6</a>] </sup>The currently supported Lucene index file format is version 2.3 (starting from ZF 1.6).</p></div>
</div>
</div>
<div class="navfooter"><table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left">
<a accesskey="p" href="zend.search.lucene.html">Prev</a> </td>
<td width="20%" align="center"><a accesskey="u" href="zend.search.lucene.html">Up</a></td>
<td width="40%" align="right"> <a accesskey="n" href="zend.search.lucene.searching.html">Next</a>
</td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Chapter 36. Zend_Search_Lucene </td>
<td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
<td width="40%" align="right" valign="top"> 36.3. Searching an Index</td>
</tr>
</table></div>
<div class="revinfo"></div>
</body>
</html>
